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Abstract — It is shown that a minimal upper estimate for the 
error probability of the formal neuron, when the latter is used as 
a restoring (decision) element, can be obtained by the Laplace 
transform of the convolution of functions as well as by means of 
the generating function of the factorial moment of the sum of 
independent random variables. It is proved that in both cases the 
obtained minimal upper estimates are absolutely identical. 

Index Terms — generating function, probability of signal 
restoration error, restoring neuron, upper estimate. 

I. Introduction 

Let us consider the formal neuron, to the inputs of which 
different versions X I ,X 2 ,...,X n ,X n+I of one and the same 
random binary signal X arrive via the binary channels 
B t ,B 2 ,...,B n ,B n+l with different error probabilities 

q i {i = \,n J t)\, and the neuron must restore the correct input 
signal X or, in other words, make a decision Y using the 
versions X 1 ,X 2 ,...,X n ,X n+1 . When the binary signal X 
arrives at the inputs of the restoring element via the channels 
of equal reliability, the decision-making, in which some value 
prevails among the signal versions, i.e. the decision-making 
by the majority principle, was for the first time described by 
J. von Neumann [1], and later V. I. Varshavski [2] 
generalized this principle to redundant analog systems. 

In the case of input channels with different reliabilities, 
adaptation of the formal neuron is needed in order to restore 
the correct signal. Adaptation is interpreted as the control 
process of weights a i (V = 1 ,« + 1 j of the neuron inputs, 
which makes these weights match the current probabilities 
q t [i = \,n+\\ of the input channels. The purpose of this 

control is to make inputs of high reliability to exert more 
influence on decision-making (i.e. on the restoration of the 
correct signal) as compared with inputs of low reliability. 
Restoration is carried out by vote-weighting by the relation 
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where 


sgn a i X i | = sgn Z , 

(1) 

Z = f>,Jf,. . 

(2) 


Both the input signal X and its versions X j (i = 1 ,n) are 
considered as binary random variables coded by the logical 
values (+1) and (—1) . It is formally assumed that the 
threshold 0 of the restoring neuron is introduced into 
consideration by means of the identity 0 = a n+l , where 
(— oo < a n+1 < oo ) and the signal X n+] = - 1 . The main 
point of this formalism is that the signal X n+i = — 1 is 


sgn Z = l 


(3) 


dumped from some imaginary binary input B n+l for any value 
of the input signal X , whereas the value q n+l is the a priori 
probability of occurrence of the signal X = +1 or, which is 
the same, the error probability of the channel B n+l . Quite a 
vast literature [3]-[7] is dedicated to threshold logic which 
takes into consideration the varying reliability of channels, 
but in this paper we express our viewpoint in the spirit of the 
ideas of W. Pierce [8], 

Let us further assume that 

J-l ifZ<0 
j+1 if Z > 0 

When Z = 0 , the solution Y at the output of the restoring 
formal neuron has the form +1 according to (3). The 
probability that the restored value Y of the signal X is not 
correct is expressed by the formula 

Q = Prob {Y * X) = Prob {>;<()}. (4) 

Here rj = XZ is a discrete random variable with 
probability distribution density / (v) . This variable is the 
sum of independent discrete variables tj i = a i XX i , and the 
function /'(y) describes the probability distribution density 
of individual summands rj i . For the realizations of random 


variables 77 and r/. we introduce the symbols v and y. , 
respectively. 

It is easy to observe that the variable r/ i takes the values 
+o, and -a, with probabilities 1 — q l and q s , respectively. 
Therefore, if we use the Dirac delta function 8{t ) , then the 
probability density f] (y ) can be represented as follows 
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f ( v i ) = (1 - </, )6(y, — a,) + q,S(v i + a , ) 
y = +a i ,—a i 
i = !,« + ! 


( 5 ) 


Such formalism is completely justified and frequently used 
due to the following two properties of the delta-function 
8(t) > 0, Vf e R 

J~<y(f>ft=i 

However / ; '(y) can also be represented as 


v,. = +a i5 - a t 
/ = 1,77+1 


( 6 ) 


The random variable 77 is the sum of independent discrete 


random variables 77, . Its distribution density f(v) can be 
defined in the form of convolution of probability distribution 
densities of summands f ] (y ) : 


./■(v) = "£'./:(v ; ) , (7) 

where * (superposition of the addition and multiplication 
signs) is the convolution symbol. 

It is obvious that in view of formula (7) the error 
probability at the decision element output can be written in 
two equivalent forms 

Q = Prob(v < 0) = J f(v)dv = J "*/;(y)dv = 

(8) 

= J "*' [Q-q, )<5(y -a,)+ q,S( Vi + a, )] dv 
and 

e=E/( v )=E"*!^( v ')= 

v<o v«> - (9) 

where the probabibty distribution density f t (v t ) is defined by 
(5) in the first case and by (6) in the second case. Integration 
or summation in both cases is carried out continuously or 
discretely over all negative values of the variable v . 
Formulas (8) and (9) give an exact value of the error 
probability of restoration of a binary signal by the formal 
neuron. 

Note that for practical calculations, formula (9) can be 
written in a more convenient form. Indeed, the complete 
number of discrete values of the variable v is 2" +1 since 


v = a l +a 2 - 1 Ya n +a B+1 , 

where a i is equal either to +(a f ) or to -(«,), whereas the 
proper sign of the weight a, is meant to be within the round 
brackets. 

By formula (9), to each discrete value of the sum v there 
corresponds the term g, (J = 1, 2" +1 ) which is the product of 
(n + 1) co-factors of the form q k or (1 — q k ). 

In particular 


Qj = /(v) = .* f i (v i ) = q 1 q 2 - - q„ q „ +l 

j=vr* 

where 

~ = k* if v k =-(a k ) 

^ tl-fc ^ v k =+(a k ) 

for any k {k. = 1 , n + 1 j . 

Thus formula (9) can also be written in the form 

G = E2/ = E?i ^ „+i> ( 10 ) 

which is more adequate for cognitive perception and practical 
realization. 

II. FINDING A MINIMAL UPPER ESTIMATE BY THE 
FIRST METHOD 
From the expression 

Q = Prob(?7 < 0) = J / ( v)dv 

it follows that for a real positive number s (s > 0) 

Q<]e- sv f(v)dv<]e- ! V(v)dv. 

But the left-hand part of this inequality is the Laplace 
transform of the function / (v) 

Z\f(v)\=]e- sv f{v)dv, 

where £ is the Laplace transform operator. 

Therefore 

QZ£[f(v)l (11) 

The random value 77 with realizations v is the sum of 
independent random variables 77. having realizations v f . In 
that case, as is known, the Laplace transform for the 
convolution /(v) of functions / (y ) is equal to the product 
of Laplace transforms of convoluted functions: 

£[/(v)]=n£[/,(v,)]. 

The latter implies that 

Q< fpMCv,)]. (12) 

By expression (5) for functions f] ( y ) and the Laplace 
transform definition, we obtain 

£[f,(v l )\=]e- s ' , ‘Z(v i )dv l = 

= jf e- sv ' [(\-q i )S(v i -a i ) + q,S(v i +a t )]dv t 

Using this expression in formula (12) we have 

e<f| j ^ [(l-^)^(y -fl,.) + ^(y +fl,.)]dy. (13) 

Here we should make use of one more property of the Dirac 
delta function 

] g(t)S(t-t 0 )dt = g(t 0 ). 
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With this property taken into account, from formula (13) 
we obtain 


(14) 


Q±m- q ‘ + q ‘ e+a,s ]• 


Here s , as mentioned above, is an arbitrary real positive 
number. Before we continue simplifying the right-hand part of 
inequality (14), we have to define a value of s for which 
expression (14) gives a minimal upper estimate. 

Passing to the natural logarithm of inequality (14) we come 
to the expression 

Infi ^ !>[(l-? f K a ' s +q t e +v '] 

Let us define here partial derivatives with respect to 
arguments a, by using the elementary fact that 

y4=rwv'» 

ax 

if y = e f(x) , and also the fact that —In f(x) = f\x) — - — . 

dx f(x) 

Hence we obtain 


Sin £ < y 1_ 

3a, ~it[{l-q,)e- v +q t e+«'] 


{ sq,e 


-s{ \-qi)e- a ‘ s \ 


or, which is the same, 


IfnlZft 


s = — In 


If the weights a, of the neuron inputs are put into 
correspondence with error probabilities q t of these inputs by 
the relations 

1-9, 


fl; = In — 

<h 

then the sought value of 5 will be 
1 


i/4 =exp [ ta (®] =exp K tai ?] 

Besides, we denote 
Then we have 

q,e 2 +(1 -q,)e 1 = l^jqfl-qf • 6 +e . 


The second co-factor in the right-hand part of this 
expression is the hypebolic cosine of the argument A, : 


For the right-hand part of this inequality to be equal to 
zero, it suffices that the following condition be fulfilled 
sq i e ra,s - si\- q.)e~ a,s = 0, 
whence it follows that 

+2 a,s _ l~g,- 


q t e 2 +i\-q t )e 2 = 2^.(1-?,.) -ch^ = 

- q , (1 — q,)' ch ^ 

Finally, for estimate (14) we can write 


e<nW?,a-?,)-ch 


(15) 


<h 


(16) 

Using equality (16) in formula (14), we obtain a minimal 
upper estimate for the error probability Q of the restoring 
neuron. Indeed, for the right-hand part of expression (14) the 
following chain of identical transforms is valid: 

2 yjqiQ-q,) 


= lyjqfl — q,) ■ 


For the error probability Q , the right-hand part of the 
above inequality is the upper estimate Q + : 

a ; - lr — 

Q =U 2ylq i (l-q i ) - ch 


The minimum Q^ mm of this upper estimate Q + is equal to 

= fl [zv?, (i-?,-)]= 2" +i • n [fo (I • - Qi ) ] • (i7) 

It is attained when for the zero argument the hyperbolic 
cosine attains the minimum equal to 1 . 

This estimate confirms in a certain sense the advantage of 
the choice of weights of the restoring neuron in compliance 
with the error probabilities of input signals according to the 
following relations 

a.=lni^l 

q i [• 

Z = 1,M + 1 


Let us take into account here that 
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m. OBTAINING A MINIMAL UPPER ESTIMATE BY 
THE SECOND METHOD 

Simulteously, for the probability Q it is useful to obtain a 
minimal upper estimate in the closed analytic form by one 
more new approach. 

As is known [9], the generating function j v (S) of the 
factorial moment of the sum 77 of independent random 
variables 77 , is equal to the product of generating functions 
7 V (.S') of the factorial moments of individual summands, i.e. 


where 

q;=o-vm+^ 

and, along with this, 

w i = S a ‘ , 0 < w ; . < go, (7 = 1,77 + 1). (25) 

Now we can find the minimum gf, of expression (24) and 
the value w m of w i will attach a minimum to the upper 
estimate of the error probability Q* of the restoring neuron. 
For this, we use the conditions 


where 


n(s)=fKtf)> 

(18) 

^ = 0 
d w t 

y v (S)=M[S"] = XS v -/(v), 

( 19 > Hence 

/ = 1 , 77 + 1 J 

y Vj (A)=M[A’'] = X^-y;(v l ) 

(20) 



(26) 


7 = 1,77 + 1 

Here M is the mathematical expectation symbol and S 
is an arbitrary complex number for which series (19) and ( 20 ) 
exist on some segment of the real axis containing the point 
S = 1. 

Since in relation (20), summation is carried out on the set 
of two possible values +a l and —a l of the variable v, , using 
( 6 ) we have 

y Vi (S) = (]-q i )S a - +q t S- a 
7 = 1,77 + 1 

The substitution of (21) into relation (18) gives 


If (26) is substituted into expression (24), then by the 
second method for a minimal upper estimate of the error 
probability of the restoring neuron we obtain the relation 

QL^.f\y qi a- q ^ ( 27 ) 

which coincides with result (17) obtained by the first method. 

The weights a, (i = 1,77 + 1) which match the error 
probabilities q t (i = 1 , 77 + lj are defined from relations (26) 
with notation (25) taken into account: 

a i= _^: U JkL 

21n S 1 -q, . 

7 = 1,77 + 1 



When v < 0 , the value S v satisfies the condition 



if of course 

0 < S' < 1. (22) 

Let us assume that inequality (22) is fulfilled. Then the 
following relation is valid: 

e=X/(v)<2>'’ /(v). 

Since every summand S v • / (v) is non-negative, we have the 
inequality 

Xsy(v)<2X/(v). 

Therefore 

Q<n(S). (23) 

The right-hand part of this expression can be taken as the 
upper estimate Q* of the error probability Q of the restoring 
neuron 

e + =n[(i-7,)^+7,^]. 

The latter relation is easily rewritten in the equivalent form 
Q + = ft Qt = n [(! ■ - 9, H + ^ j , (24) 


Since the value S satisfies condition (22), we have In S < 0 
and therefore 


a, =W.fat3t 

q t , 


7 = 1,77 + 1 


(28) 


where 

K - 1 

(29) 

0 < A" < t» J 

Thus, the weights a i (i = 1,77 + 1) , which are consistent with 
the error probabilities q t [i = 1, n + 1 j and attach a minimum 

to the upper estimate of the error probability of the restoring 
neuron, are defined to within the a general positive factor K . 


IV. Conclusion 

A minimal upper estimate of the error probability of the 
restoring formal neuron is defined by formula (17) or, which 
is the same, by formula (27). In both cases the result can be 
written in the form 

QL = ex P^-Z^te)], ( 3 °) 
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where 

A(q t ) - |ln [2^/9,- (1 - q t ) ]| . (31) 

In view of relations (31) confirming the non-negativity of 
the values A{q t ) , formula (30) implies that an increase of 
the number n of inputs of the formal decision neuron bings 
about a monotone decrease of the minimal upper estimate of 
the error probability of restoration of the binary signal Qf n 
by the exponential law if, certainly, the error probabilities 

q. {i = 1 , « + 1 j at these inputs are not equal to when the 


minimal upper estimate of the error probability is equal 
to 1. 

This result is demonstrates an essenatial inner connection 
with Shannon’s theorem [10]. According to this theorem, the 
number of messages of length n (duration r ) composed of 
individual symbols - both in the absence and in the presence 
of fixed and probabilistic constraints (in the latter case it is 
assumed that the source is ergodic) - grows by the 
asymptotically exponential law as n (or r ) increases. In 
particular we understand this connection as follows: as the 
number n of inputs of the restoring formal neuron increases, 
the initial information to be used in making the decision Y 
increases by the exponential law if there are a number of 
possible versions of the input signal, while the minimal upper 
estimate gf, of the probability Q that the made decision is 
erroneous decreases by the same exponential law. 
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